Optical array sensor for use with autonomous vehicle control systems

ABSTRACT

A downwardly-directed optical array sensor may be used in an autonomous vehicle to enable a velocity (e.g., an overall velocity having a direction and magnitude, or a velocity in a particular direction, e.g., along a longitudinal or lateral axis of a vehicle) to be determined based upon images of a ground or driving surface captured from multiple downwardly-directed optical sensors having different respective fields of view.

BACKGROUND

As computing and vehicular technologies continue to evolve,autonomy-related features have become more powerful and widelyavailable, and capable of controlling vehicles in a wider variety ofcircumstances. For automobiles, for example, the automotive industry hasgenerally adopted SAE International standard J3016, which designates 6levels of autonomy. A vehicle with no autonomy is designated as Level 0,and with Level 1 autonomy, a vehicle controls steering or speed (but notboth), leaving the operator to perform most vehicle functions. WithLevel 2 autonomy, a vehicle is capable of controlling steering, speedand braking in limited circumstances (e.g., while traveling along ahighway), but the operator is still required to remain alert and beready to take over operation at any instant, as well as to handle anymaneuvers such as changing lanes or turning. Starting with Level 3autonomy, a vehicle can manage most operating variables, includingmonitoring the surrounding environment, but an operator is stillrequired to remain alert and take over whenever a scenario the vehicleis unable to handle is encountered. Level 4 autonomy provides an abilityto operate without operator input, but only in specific conditions suchas only certain types of roads (e.g., highways) or only certaingeographical areas (e.g., specific cities for which adequate mappingdata exists). Finally, Level 5 autonomy represents a level of autonomywhere a vehicle is capable of operating free of operator control underany circumstances where a human operator could also operate.

A fundamental challenge of any autonomy-related technology relates tocollecting and interpreting information about a vehicle's surroundingenvironment, along with making and implementing decisions toappropriately control the vehicle given the current environment withinwhich the vehicle is operating. Therefore, continuing efforts are beingmade to improve each of these aspects, and by doing so, autonomousvehicles increasingly are able to reliably handle a wider variety ofsituations and accommodate both expected and unexpected conditionswithin an environment.

SUMMARY

The present disclosure is directed in part to a downwardly-directedoptical array sensor that may be used in an autonomous vehicle to enablea velocity (e.g., an overall velocity having a direction and magnitude,or a velocity in a particular direction, e.g., along a longitudinal orlateral axis of a vehicle) to be determined based upon images of aground or driving surface captured from multiple downwardly-directedoptical sensors having different respective fields of view.

Therefore, consistent with one aspect of the invention, a method ofautonomously operating a vehicle may include receiving a plurality ofimages of a ground surface captured from a plurality ofdownwardly-directed optical sensors mounted to the vehicle, each of theplurality of downwardly-directed optical sensors having a respectivefield of view, and the plurality of downwardly-directed optical sensorsarranged such that the respective fields of view differ from oneanother, processing the plurality of images using at least one processorto determine a velocity of the vehicle, and autonomously controllingmovement of the vehicle using the determined velocity.

In some embodiments, at least a subset of the plurality ofdownwardly-directed optical sensors are arranged to have respectivefields of view that are positionally offset along a one-dimensionalarray. In addition, in some embodiments, at least a subset of theplurality of downwardly-directed optical sensors are arranged to haverespective fields of view that are positionally offset in atwo-dimensional array. Moreover, in some embodiments, theone-dimensional array extends generally along a longitudinal axis of thevehicle. Also, in some embodiments, the respective fields of view of atleast a subset of the plurality of downwardly-directed optical sensorspartially overlap with one another.

In some embodiments, processing the plurality of images includescorrelating first and second images respectively captured at first andsecond times by a first downwardly-directed optical sensor among theplurality of downwardly-directed optical sensors to determine apositional displacement of the vehicle between the first and secondtimes, and determining the velocity based upon the determined positionaldisplacement of the vehicle between the first and second times.

Also, in some embodiments, the subset of downwardly-directed opticalsensors arranged to have respective fields of view that are positionallyoffset along the one-dimensional array includes first and seconddownwardly-directed optical sensors, and processing the plurality ofimages includes correlating a first image captured at a first time bythe first downwardly-directed optical sensor with a second imagecaptured at a second time by the second downwardly-directed opticalsensor to determine a positional displacement of the vehicle between thefirst and second times, and determining the velocity based upon thedetermined positional displacement of the vehicle between the first andsecond times.

Moreover, in some embodiments, the subset of downwardly-directed opticalsensors arranged to have respective fields of view that are positionallyoffset along the one-dimensional array includes first and seconddownwardly-directed optical sensors, and processing the plurality ofimages includes performing a first correlation between a first imagecaptured at a first time by the first downwardly-directed optical sensorand a second image captured at a second time by the firstdownwardly-directed optical sensor, performing a second correlationbetween the first image captured at the first time by the firstdownwardly-directed optical sensor and a third image captured at a thirdtime by the second downwardly-directed optical sensor, using one of thefirst and second correlations to determine a positional displacement ofthe vehicle between the first time and one of the second and thirdtimes, and determining the velocity based upon the determined positionaldisplacement. In some embodiments, the second and third times aresubstantially the same. Moreover, in some embodiments, using the one ofthe first and second correlations to determine the positionaldisplacement includes using the first correlation in response todetermining that the second image could be correlated to the firstimage, and using the second correlation in response to determining thatthe third could be correlated to the first image.

In some embodiments, the subset of downwardly-directed optical sensorsarranged to have respective fields of view that are positionally offsetalong the one-dimensional array includes more than twodownwardly-directed optical sensors. Further, in some embodiments, theone-dimensional array extends generally along a longitudinal axis of thevehicle, and the subset of downwardly-directed optical sensors arrangedto have respective fields of view that are positionally offset along theone-dimensional array collectively provide an effective aperture that islarger along the longitudinal axis of the vehicle than along a lateralaxis of the vehicle.

Moreover, in some embodiments, the plurality of images includes imagescaptured at a plurality of times by each of the subset ofdownwardly-directed optical sensors. Also, in some embodiments,processing the plurality of images includes stitching together multipleimages from each of the plurality of times to generate a composite imagefor each of the plurality of times, correlating a first composite imagefor a first time among the plurality of times with a second compositeimage for a second time among the plurality of times to determine apositional displacement of the vehicle between the first and secondtimes, and determining the velocity based upon the determined positionaldisplacement of the vehicle between the first and second times. In someembodiments, the respective fields of view of the subset ofdownwardly-directed optical sensors partially overlap with one another,and stitching together the multiple images from each of the plurality oftimes includes correlating overlapping regions of the multiple images.Also, in some embodiments, the plurality of times are within a singlecontrol cycle among a plurality of control cycles.

Moreover, in some embodiments, the plurality of downwardly-directedoptical sensors are coupled to a primary vehicle control system of thevehicle, the primary vehicle control system includes the at least oneprocessor, and receiving the plurality of images, processing theplurality of images, and autonomously controlling movement of thevehicle are performed by the primary vehicle control system.

Also, in some embodiments, the plurality of downwardly-directed opticalsensors are coupled to a secondary vehicle control system of thevehicle, the secondary vehicle control system includes the at least oneprocessor, and receiving the plurality of images, processing theplurality of images, and autonomously controlling movement of thevehicle are performed by the second vehicle control system subsequent todetection of an adverse event for a primary vehicle control system ofthe vehicle.

Further, in some embodiments, the plurality of downwardly-directedoptical sensors are disposed on an undercarriage of the vehicle. Also,in some embodiments, determining the velocity includes determining alateral velocity of the vehicle. Moreover, in some embodiments,determining the velocity includes determining a longitudinal velocity ofthe vehicle. In some embodiments, determining the velocity includesdetermining a magnitude component and a direction component of thevelocity.

Moreover, in some embodiments, the vehicle includes a fully autonomousvehicle. Further, in some embodiments, the vehicle includes anautonomous wheeled vehicle. In addition, in some embodiments, thevehicle includes an autonomous automobile, bus or truck. Also, in someembodiments, the plurality of downwardly-directed optical sensors areinfrared sensors. Some embodiments may further include capturing theplurality of images with the plurality of downwardly-directed opticalsensors, and illuminating the ground surface with a strobe emitter whencapturing the plurality of images.

Consistent with another aspect of the invention, a vehicle controlsystem may include a plurality of downwardly-directed optical sensorsconfigured to be mounted to a vehicle to capture images of a groundsurface, each of the plurality of downwardly-directed optical sensorshaving a respective field of view, and the plurality ofdownwardly-directed optical sensors arranged such that the respectivefields of view differ from one another, and at least one processorcoupled to the plurality of downwardly-directed optical sensors, the atleast one processor configured to receive a plurality of images of theground surface captured from the plurality of downwardly-directedoptical sensors, process the plurality of images to determine a velocityof the vehicle, and autonomously control movement of the vehicle usingthe determined velocity.

Consistent with another aspect of the invention, a vehicle may include aplurality of downwardly-directed optical sensors configured to bemounted to a vehicle to capture images of a ground surface, each of theplurality of downwardly-directed optical sensors having a respectivefield of view, and the plurality of downwardly-directed optical sensorsarranged such that the respective fields of view differ from oneanother, and at least one processor coupled to the plurality ofdownwardly-directed optical sensors, the at least one processorconfigured to receive a plurality of images of the ground surfacecaptured from the plurality of downwardly-directed optical sensors,process the plurality of images to determine a velocity of the vehicle,and autonomously control movement of the vehicle using the determinedvelocity.

Consistent with another aspect of the invention, a method ofautonomously operating a vehicle may include receiving a plurality ofimages of a ground surface captured from a plurality ofdownwardly-directed optical sensors mounted to the vehicle, each of theplurality of downwardly-directed optical sensors having a respectivefield of view, and the plurality of downwardly-directed optical sensorsarranged such that the respective fields of view differ from oneanother, processing the plurality of images using at least one processorto determine a velocity of the vehicle, and autonomously controllingmovement of the vehicle using the determined velocity.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example hardware and software environment for anautonomous vehicle.

FIG. 2 is a flowchart illustrating an example sequence of operations forcontrolling the autonomous vehicle of FIG. 1.

FIG. 3 is a functional view of an example scene, illustrating exampleprimary and controlled stop trajectories generated for the autonomousvehicle of FIG. 1.

FIG. 4 is a functional side elevation view of a one-dimensional opticalarray sensor suitable for use in the autonomous vehicle of FIG. 1.

FIG. 5 is a functional top plan view of the optical array sensor of FIG.4.

FIGS. 6A-6D functionally illustrate detection of vehicle movement by theoptical array sensor of FIGS. 4-5 between two points in time, with FIG.6A functionally illustrating the position of the vehicle at a firsttime, and with FIGS. 6B, 6C, and 6D illustrating the position of thevehicle at a second time, but traveling at three different speeds.

FIG. 7 is a functional top plan view of a two-dimensional optical arraysensor suitable for use in the autonomous vehicle of FIG. 1.

FIG. 8 is a flowchart illustrating an example sequence of operations forsensing velocity with the optical array sensor of FIGS. 4-5.

FIG. 9 is a flowchart illustrating another example sequence ofoperations for sensing velocity with the optical array sensor of FIGS.4-5.

DETAILED DESCRIPTION

The various implementations discussed hereinafter are directed toautonomous vehicle control systems and sensors for use therewith. Insome implementations, for example, controlled stop functionality may beimplemented in an autonomous vehicle using a secondary vehicle controlsystem that supplements a primary vehicle control system to perform acontrolled stop if an adverse event is detected in the primary vehiclecontrol system, and using a redundant lateral velocity determined by adifferent sensor from that used by the primary vehicle control system.In addition, in some implementations, a downwardly-directed opticalarray sensor may be used in an autonomous vehicle to enable a velocity(e.g., an overall velocity having a direction and magnitude, or avelocity in a particular direction, e.g., along a longitudinal orlateral axis of a vehicle) to be determined based upon images of aground or driving surface captured from multiple downwardly-directedoptical sensors having different respective fields of view.

Prior to a discussion of these implementations, however, an examplehardware and software environment within which the various techniquesdisclosed herein may be implemented will be discussed.

Hardware and Software Environment

Turning to the Drawings, wherein like numbers denote like partsthroughout the several views, FIG. 1 illustrates an example autonomousvehicle 100 within which the various techniques disclosed herein may beimplemented. Vehicle 100, for example, is shown driving on a road 101,and vehicle 100 may include a powertrain 102 including a prime mover 104powered by an energy source 106 and capable of providing power to adrivetrain 108, as well as a control system 110 including a directioncontrol 112, a powertrain control 114 and brake control 116. Vehicle 100may be implemented as any number of different types of land-basedvehicles, including vehicles capable of transporting people and/orcargo, and it will be appreciated that the aforementioned components102-116 can vary widely based upon the type of vehicle within whichthese components are utilized.

The implementations discussed hereinafter, for example, will focus on awheeled land vehicle such as a car, van, truck, bus, etc. In suchimplementations, the prime mover 104 may include one or more electricmotors and/or an internal combustion engine (among others), while energysource 106 may include a fuel system (e.g., providing gasoline, diesel,hydrogen, etc.), a battery system, solar panels or other renewableenergy source, a fuel cell system, etc., and drivetrain 108 may includewheels and/or tires along with a transmission and/or any othermechanical drive components suitable for converting the output of primemover 104 into vehicular motion, as well as one or more brakesconfigured to controllably stop or slow the vehicle and direction orsteering components suitable for controlling the trajectory of thevehicle (e.g., a rack and pinion steering linkage enabling one or morewheels of vehicle 100 to pivot about a generally vertical axis to varyan angle of the rotational planes of the wheels relative to thelongitudinal axis of the vehicle). In some implementations, combinationsof powertrains and energy sources may be used, e.g., in the case ofelectric/gas hybrid vehicles, and in some instances multiple electricmotors (e.g., dedicated to individual wheels or axles) may be used as aprime mover. In the case of a hydrogen fuel cell implementation, theprime mover may include one or more electric motors and the energysource may include a fuel cell system powered by hydrogen fuel.

Direction control 112 may include one or more actuators and/or sensorsfor controlling and receiving feedback from the direction or steeringcomponents to enable the vehicle to follow a desired trajectory.Powertrain control 114 may be configured to control the output ofpowertrain 102, e.g., to control the output power of prime mover 104, tocontrol a gear of a transmission in drivetrain 108, etc., therebycontrolling a speed and/or direction of the vehicle. Brake control 116may be configured to control one or more brakes that slow or stopvehicle 100, e.g., disk or drum brakes coupled to the wheels of thevehicle.

Other vehicle types, including but not limited to airplanes, spacevehicles, helicopters, drones, military vehicles, all-terrain or trackedvehicles, ships, submarines, construction equipment, etc., willnecessarily utilize different powertrains, drivetrains, energy sources,direction controls, powertrain controls and brake controls, as will beappreciated by those of ordinary skill having the benefit of the instantdisclosure. Moreover, in some implementations some of the components maybe combined, e.g., where directional control of a vehicle is primarilyhandled by varying an output of one or more prime movers. Therefore, theinvention is not limited to the particular application of theherein-described techniques in an autonomous wheeled land vehicle.

In the illustrated implementation, full or semi-autonomous control overvehicle 100 is primarily implemented in a primary vehicle control system120, which may include one or more processors 122 and one or morememories 124, with each processor 122 configured to execute program codeinstructions 126 stored in a memory 124.

A primary sensor system 130 may include various sensors suitable forcollecting information from a vehicle's surrounding environment for usein controlling the operation of the vehicle. For example, a globalpositioning system (GPS) sensor 132 may be used to determine thelocation of the vehicle on the Earth. Radio Detection And Ranging(RADAR) and Light Detection and Ranging (LIDAR) sensors 134, 136, aswell as a digital camera 138, may be used to sense stationary and movingobjects within the immediate vicinity of a vehicle. An inertialmeasurement unit (IMU) 140 may include multiple gyroscopes andaccelerometers capable of detection linear and rotational motion of avehicle in three directions.

The outputs of sensors 132-140 may be provided to a set of primarycontrol subsystems 150, including, a computer vision subsystem 152, anobstacle avoidance subsystem 154 and a navigation/guidance subsystem156. Computer vision subsystem 152 may be configured to take the inputfrom RADAR sensor 134, LIDAR sensor 136 and/or digital camera 138 todetect and identify objects surrounding the vehicle, as well as theirmotion relative to the vehicle. Obstacle avoidance subsystem 154 may usethis information to detect stationary and/or moving obstacles in thevicinity of the vehicle that should be avoided. Navigation/guidancesubsystem 156 determines a trajectory and speed for the vehicle basedupon the desired destination and path and the detected obstacles in thevicinity of the vehicle.

It will be appreciated that the collection of components illustrated inFIG. 1 for primary vehicle control system 120 is merely exemplary innature. Individual sensors may be omitted in some implementations,multiple sensors of the types illustrated in FIG. 1 may be used forredundancy and/or to cover different regions around a vehicle, and othertypes of sensors may be used. Likewise, different types and/orcombinations of control subsystems may be used in other implementations.Further, while subsystems 152-156 are illustrated as being separate fromprocessors 122 and memory 124, it will be appreciated that in someimplementations, some or all of the functionality of a subsystem 152-156may be implemented with program code instructions 126 resident in one ormore memories 124 and executed by one or more processors 122, and thatthese subsystems 152-156 may in some instances be implemented using thesame processors and/or memory. Subsystems in some implementations may beimplemented at least in part using various dedicated circuit logic,various processors, various field-programmable gate arrays (“FPGA”),various application-specific integrated circuits (“ASIC”), various realtime controllers, and the like, and as noted above, multiple subsystemsmay utilize common circuitry, processors, sensors and/or othercomponents. Further, the various components in primary vehicle controlsystem 120 may be networked in various manners.

In the illustrated implementation, vehicle 100 also includes a secondaryvehicle control system 160, which may include one or more processors 162and one or more memories 164 capable of storing instructions 166 forexecution by processor(s) 162. Secondary vehicle control system 160, aswill be discussed in greater detail below, may be used as a redundant orbackup control system for vehicle 100, and may be used, among otherpurposes, to perform controlled stops in response to adverse eventsdetected in primary vehicle control system 120.

Secondary vehicle control system 160 may also include a secondary sensorsystem 170 including various sensors used by secondary vehicle controlsystem 160 to sense the condition and/or surroundings of vehicle 100.For example, one or more wheel encoders 172 may be used to sense thevelocity of each wheel, while an IMU sensor 174 may be used to generatelinear and rotational motion information about the vehicle. In addition,and as will be discussed in greater detail below, a downwardly-directedoptical array sensor 176 may be used to sense vehicle motion relative toa ground or driving surface. Secondary vehicle control system 160 mayalso include several secondary control subsystems 180, including amonitor subsystem 182, which is used to monitor primary vehicle controlsystem 120, a velocity calculation subsystem 184, which is used tocalculate at least a lateral velocity for the vehicle, and a controlledstop subsystem 186, which is used to implement a controlled stop forvehicle 100 using the lateral velocity determined by velocitycalculation subsystem 184 upon detection of an adverse event by monitorsubsystem 182. Other sensors and/or subsystems that may be utilized insecondary vehicle control system 160, as well as other variationscapable of being implemented in other implementations, will be discussedin greater detail below.

In general, an innumerable number of different architectures, includingvarious combinations of software, hardware, circuit logic, sensors,networks, etc. may be used to implement the various componentsillustrated in FIG. 1. Each processor may be implemented, for example,as a microprocessor and each memory may represent the random accessmemory (RAM) devices comprising a main storage, as well as anysupplemental levels of memory, e.g., cache memories, non-volatile orbackup memories (e.g., programmable or flash memories), read-onlymemories, etc. In addition, each memory may be considered to includememory storage physically located elsewhere in vehicle 100, e.g., anycache memory in a processor, as well as any storage capacity used as avirtual memory, e.g., as stored on a mass storage device or on anothercomputer or controller. One or more processors illustrated in FIG. 1, orentirely separate processors, may be used to implement additionalfunctionality in vehicle 100 outside of the purposes of autonomouscontrol, e.g., to control entertainment systems, to operate doors,lights, convenience features, etc.

In addition, for additional storage, vehicle 100 may also include one ormore mass storage devices, e.g., a floppy or other removable disk drive,a hard disk drive, a direct access storage device (DASD), an opticaldrive (e.g., a CD drive, a DVD drive, etc.), a solid state storage drive(SSD), network attached storage, a storage area network, and/or a tapedrive, among others. Furthermore, vehicle 100 may include an interface190 with one or more networks (e.g., a LAN, a WAN, a wireless network,and/or the Internet, among others) to permit the communication ofinformation with other computers and electronic devices, including, forexample, a central service, such as a cloud service, from which vehicle100 receives map and other data for use in autonomous control thereof.Moreover, a user interface 192 may be provided to enable vehicle 100 toreceive a number of inputs from and generate outputs for a user oroperator, e.g., one or more displays, touchscreens, voice and/or gestureinterfaces, buttons and other tactile controls, etc. Otherwise, userinput may be received via another computer or electronic device, e.g.,via an app on a mobile device or via a web interface.

Each processor illustrated in FIG. 1, as well as various additionalcontrollers and subsystems disclosed herein, generally operates underthe control of an operating system and executes or otherwise relies uponvarious computer software applications, components, programs, objects,modules, data structures, etc., as will be described in greater detailbelow. Moreover, various applications, components, programs, objects,modules, etc. may also execute on one or more processors in anothercomputer coupled to vehicle 100 via network, e.g., in a distributed,cloud-based, or client-server computing environment, whereby theprocessing required to implement the functions of a computer program maybe allocated to multiple computers and/or services over a network.

In general, the routines executed to implement the variousimplementations described herein, whether implemented as part of anoperating system or a specific application, component, program, object,module or sequence of instructions, or even a subset thereof, will bereferred to herein as “program code.” Program code typically comprisesone or more instructions that are resident at various times in variousmemory and storage devices, and that, when read and executed by one ormore processors, perform the steps necessary to execute steps orelements embodying the various aspects of the invention. Moreover, whilethe invention has and hereinafter will be described in the context offully functioning computers and systems, it will be appreciated that thevarious implementations described herein are capable of beingdistributed as a program product in a variety of forms, and that theinvention applies equally regardless of the particular type of computerreadable media used to actually carry out the distribution. Examples ofcomputer readable media include tangible, non-transitory media such asvolatile and non-volatile memory devices, floppy and other removabledisks, solid state drives, hard disk drives, magnetic tape, and opticaldisks (e.g., CD-ROMs, DVDs, etc.), among others.

In addition, various program code described hereinafter may beidentified based upon the application within which it is implemented ina specific implementation. However, it should be appreciated that anyparticular program nomenclature that follows is used merely forconvenience, and thus the invention should not be limited to use solelyin any specific application identified and/or implied by suchnomenclature. Furthermore, given the typically endless number of mannersin which computer programs may be organized into routines, procedures,methods, modules, objects, and the like, as well as the various mannersin which program functionality may be allocated among various softwarelayers that are resident within a typical computer (e.g., operatingsystems, libraries, API's, applications, applets, etc.), it should beappreciated that the invention is not limited to the specificorganization and allocation of program functionality described herein.

Those skilled in the art will recognize that the exemplary environmentillustrated in FIG. 1 is not intended to limit the present invention.Indeed, those skilled in the art will recognize that other alternativehardware and/or software environments may be used without departing fromthe scope of the invention.

Lateral Velocity Determinations for Controlled Stops

As noted above, a fundamental challenge of any autonomy-relatedtechnology relates to collecting and interpreting information about avehicle's surroundings, along with making and implementing decisions toappropriately control the vehicle given the current environment withinwhich the vehicle is operating. Continuing efforts are being made toimprove each of these aspects, and by doing so, autonomous vehiclesincreasingly are able to reliably handle a wider variety of situationsand handle more unknown or unexpected events.

Despite these advances, however, autonomous vehicle control systems, aswith practically all electronic systems, are not completely immune fromhardware and/or software failures from time to time. Sensors can fail,as can processors, power supplies, and other hardware components, andsoftware can sometimes hang or otherwise cause a hardware system tobecome unresponsive. These various types of adverse events can, in someinstances, lead to a partial or full inability of a vehicle controlsystem to control a vehicle in a desired fashion.

It has been proposed, for example, for autonomous vehicles to support“controlled stop” functionality, whereby a vehicle will perform acontrolled maneuver to bring the vehicle to a stopped conditionregardless of any adverse event in the vehicle. A controlled stop ofthis nature generally requires, at the least, an ability to sense thevelocity of the vehicle. A wide variety of existing sensors cangenerally sense longitudinal velocity (i.e., velocity along alongitudinal axis of the vehicle, generally the speed of the vehicle inits primary direction of movement); however, lateral velocity (i.e., thespeed of the vehicle in a direction generally transverse to the primarydirection of movement) can be more difficult to sense and/or determine,particularly without the use of more sophisticated sensors than can beused for determining longitudinal velocity. Moreover, lateral velocityis generally much smaller than longitudinal velocity, but still even arelatively small lateral velocity can result in a vehicle not followinga desired path, and potentially causing a vehicle to stray into adjacentlanes or obstacles during a controlled stop. As an example, wheelencoders may be used to estimate longitudinal velocity; however, anymisalignment or tilt of the wheels may cause a vehicle to changeorientation and thus depart from a desired path. Therefore, a needexists for a manner of determining lateral velocity in connection withperforming a controlled stop.

Thus, in some implementations consistent with the invention, one or moreadditional sensors that are separate from a primary lateral velocitysensor of a primary vehicle control system for a vehicle may be utilizedto determine a lateral velocity for the vehicle that may be used by asecondary vehicle control system to control the vehicle in response toan adverse event detected in a primary vehicle control system (e.g., ahardware and/or software failure in the primary vehicle control system,a failure in one or more of the primary sensors, a power supply failurein the primary vehicle control system, etc.). The control by thesecondary control system may be used, for example, to execute acontrolled stop of the vehicle, and thus bring the vehicle to a stop ifthe primary vehicle control system is not fully capable of operating thevehicle. In some implementations therefore an autonomous vehicle may beoperated by a secondary vehicle control system having access to one ormore additional sensors that are different from the primary lateralvelocity sensor of the vehicle yet may be used to determine at leastlateral velocity.

A primary lateral velocity sensor, in this regard, may be considered toinclude any type of sensor used by a primary vehicle control system tosense or otherwise determine the lateral velocity of the vehicle. Aprimary lateral velocity sensor may, in some instances, be a sensor thatis exclusively dedicated to sensing lateral velocity, while in otherinstances, a primary lateral velocity sensor may sense lateral velocityalong with additional vehicle parameters. In the illustratedimplementation, for example, a LIDAR sensor such as LIDAR sensor maydetermine lateral velocity among a wide variety of other vehicleparameters in connection with determining a pose of the vehicle.

In addition, the aforementioned one or more additional sensors mayinclude various types of sensors capable of sensing lateral velocity. Inthis regard, the lateral velocity determined using the one or moreadditional sensors is referred to herein as a redundant lateral velocityinsofar as the redundant lateral velocity is separately determined fromthe lateral velocity determined from the primary lateral velocitysensor. As with the primary lateral velocity sensor, any of suchadditional sensors may also be capable of determining additional vehicleparameters beyond lateral velocity (e.g., longitudinal velocity,acceleration, etc.). Furthermore, a secondary vehicle control system mayalso rely on other sensors when performing a controlled stop in someinstances, including, for example, wheel encoders, IMUs, etc., fordetermining longitudinal velocity. As a result, the secondary vehiclecontrol system may, in some instances, execute a controlled stopindependent of the primary vehicle control system.

It will be appreciated that lateral velocity may be determined andrepresented in a number of manners consistent with the invention. Forexample, lateral velocity may be represented in some instances by amagnitude and direction (which may be indicated by a positive ornegative magnitude) of the vehicle velocity along a lateral axis of thevehicle (i.e., the axis that is orthogonal to a longitudinal axis of thevehicle representing the axis along which the vehicle travels whentraveling in a straight line). Alternatively, the lateral velocity maybe represented as a lateral component of an overall velocity of thevehicle represented by an angular direction and magnitude along thatangular direction. Thus, a sensor that senses lateral velocity in someimplementations is not necessarily limited to sensing only velocityalong a lateral axis, but may instead be a sensor that senses an overallvelocity (i.e., direction/heading and magnitude/speed) from which alateral component thereof may be determined.

In various implementations, the primary and secondary vehicle controlsystems may be completely independent from one another in terms of bothhardware and software, e.g., as illustrated by primary and secondaryvehicle control systems 120, 160 in FIG. 1, to provide for completelyindependent operation of secondary vehicle control system 160 inresponse to an adverse event associated with primary vehicle controlsystem 120. Further, in some implementations, the secondary vehiclecontrol system may be limited to functionality for performing controlledstops, e.g., in terms of being a functionally-limited, lesssophisticated, and less expensive backup vehicle control system.However, the invention is not limited to such implementations.

For example, in some implementations, secondary vehicle control systemmay have comparable functionality to primary vehicle control system, orin the least may have additional control functionality beyond performingcontrolled stops, such that the secondary vehicle control systemoperates as a redundant vehicle control system or otherwise is capableof taking over control of a vehicle to resume operations previouslycontrolled by the primary vehicle control system, and potentiallywithout any perceptible loss of control or functionality. Thus, asecondary vehicle control system can have a wide range of functionalityin different implementations.

Moreover, while primary and secondary vehicle control systems 120, 160of FIG. 1 are illustrated as utilizing separate hardware, software andsensors, in other implementations primary and secondary vehicle controlsystems can share some hardware, software and/or sensors, so long as thesecondary vehicle control system is still capable of initiating acontrolled stop in response to an adverse event in the primary vehiclecontrol system that inhibits the primary vehicle control system fromcontinuing to control the vehicle in a desirable manner.

For example, one or more primary and/or secondary sensors may be sharedby (or otherwise in communication with) both a primary and secondaryvehicle control system in some implementations. Thus, for example, insome implementations, at least some of the secondary sensors may beaccessible to the primary vehicle control system, e.g., for verifying orcross-checking vehicle parameters calculated by different sensors, forusing secondary vehicle control system sensors in the primary control ofthe vehicle, for monitoring the status of sensors, or for serving asredundant sensors capable of being used when other sensors of the sametype fail, etc. Furthermore, in some implementations, one or moresensors may not be dedicated to the primary or secondary vehicle controlsystem, but may be separate from both systems yet accessible to one orboth of the systems. Furthermore, from the perspective of an additionalsensor from which lateral velocity may be determined in connection withperforming a controlled stop, the additional sensor may be any sensorthat may or may not be accessible to the primary vehicle control system(whether a primary sensor, a secondary sensor, or other sensor), butthat is different from the sensor that is used by the primary vehiclecontrol system to determine lateral velocity during normal operation ofthe primary vehicle control system.

Likewise, it will be appreciated that some or all of the other hardwareutilized by a primary or secondary vehicle control system may beutilized for both control systems. For example, in some implementationsthe primary and secondary vehicle control systems may share one or moreprocessors, memories, mass storage devices, network interfaces,housings, power supplies, software, etc. In one example implementation,for example, primary and secondary vehicle control systems may beexecuted on the same processors, with the secondary vehicle controlsystem configured within lower level software capable of implementing acontrolled stop in response to detected unresponsiveness by a higherlevel primary vehicle control system. In other implementations, theprimary and secondary vehicle control systems may execute on differentprocessors but share other hardware components. Therefore, the inventionis not limited to the particular configuration of primary and secondaryvehicle control systems 120, 160 illustrated in FIG. 1.

Now turning to FIG. 2, this figure illustrates an example sequence ofoperations 200 for use in controlling a vehicle using the primary andsecondary vehicle control systems 120, 160 of FIG. 1. Blocks 202-206,for example, illustrate at a very high level an iterative looprepresenting the operation of navigation/guidance subsystem 156 ofprimary vehicle control system 120. In block 202, navigation/guidancesubsystem 156 determines primary and controlled stop trajectories forthe vehicle. The primary trajectory represents the desired path of thevehicle over a relatively brief time period, taking into account, forexample, the desired destination and route of the vehicle, the immediatesurroundings of the vehicle, and any obstacles or other objects in theimmediate surroundings, and it is this trajectory that the primaryvehicle control system uses to control the vehicle in block 204.

The controlled stop trajectory, in contrast, represents an alternatetrajectory for the vehicle that may be undertaken in response to anadverse event in the primary vehicle control system. Depending upon thesurroundings, the controlled stop trajectory may, for example, directthe vehicle onto the shoulder of a highway, into the parking lane of acity street, or to another area outside of regular traffic flow.Alternatively, the controlled trajectory may bring the vehicle to a stopwhile continuing in the same lane or path as the primary trajectory.

It will be appreciated that blocks 202 and 204 may be implemented in awide variety of manners in various implementations, and thatimplementing the controlled stop functionality described herein inconnection with those different manners would be well within theabilities of those of ordinary skill having the benefit of the instantdisclosure.

Block 206 periodically communicates the determined controlled stoptrajectory to monitor subsystem 182 of secondary vehicle control system160. In the illustrated implementation, the trajectory is communicatedvia a trajectory message that additionally functions as a heartbeatmessage from primary vehicle control system 120. Additional informationmay also be communicated in a trajectory message, e.g., the occurrenceof any adverse events, a timestamp, or any other information that may beuseful for secondary vehicle control system. In other implementations, atrajectory may be stored in a shared memory, while other implementationsmay utilize other manners to make a determined trajectory available foraccess by the secondary vehicle control system.

Thus, primary vehicle control system 120 may continuously update acontrolled stop trajectory for the vehicle based upon the vehicle'scurrent surroundings and status, and provide regular updates to thesecondary vehicle control system such that the secondary vehicle controlsystem may assume control and implement a controlled stop operation inthe event of an adverse event. Consequently, primary vehicle controlsystem 120 may adjust a controlled stop trajectory depending uponvehicle conditions, e.g., to change the controlled stop trajectory whena pedestrian or another vehicle is a parking lane or highway shoulder.

It will be appreciated that in many instances determining a vehicletrajectory based upon a vehicle's current surroundings is generally acomputationally intensive operation that relies on large volumes of datafrom various complex sensors. Thus, by determining the controlled stoptrajectory in the primary vehicle control system, the computationalresources and sensors that may be required for trajectory determinationmay be omitted from a secondary vehicle control system in someimplementations. In other implementations, however, a controlled stoptrajectory may be determined in the secondary vehicle control system,and such functionality may be omitted from primary vehicle controlsystem 120.

Monitor subsystem 182 of secondary vehicle control system 160 iteratesbetween blocks 208-214. In block 208, monitor subsystem 182 waits for anext trajectory message from the primary vehicle control system. Inaddition, a watchdog timer runs to ensure that trajectory messages arereceived within a required interval, as the failure to receive atrajectory message within an interval may be indicative of an adverseevent in the primary vehicle control system. Upon receipt of a message,or after expiration of the watchdog timer without receiving a message,control passes to block 210 to determine if a message was received. Ifso, control passes to block 212 to determine if the received messageincludes any indication of an adverse event requiring that the secondaryvehicle control system assume control of the vehicle to initiate acontrolled stop. If not, control passes to block 214 to store theupdated controlled stop trajectory provided in the message, and controlreturns to block 208 to start the watchdog timer and wait for the nexttrajectory message. Thus, during normal operation of primary vehiclecontrol system 120, monitor subsystem 182 maintains an up to datecontrolled stop trajectory for use if an adverse event ever occurs.

If, however, an adverse event is detected, either as a result of afailure to receive a trajectory message within the required interval, oras a result of an adverse event being signaled in the trajectorymessage, blocks 210 and 212 will instead notify controlled stopsubsystem 186 of the need to initiate a controlled stop of the vehicle.In block 216, subsystem 186 may optionally generate a controlled stopalert, e.g., by displaying information on a vehicle display, generatingaudible and/or visual alerts, or otherwise indicating that a controlledstop operation is being initiated. In other implementations, however, noalert may be generated.

Block 218 next retrieves the last-stored controlled stop trajectory, andblock 220 then implements that trajectory while monitoring the velocityof the vehicle. The velocity in the illustrated implementation is basedupon longitudinal and lateral velocities determined by velocitycalculation subsystem 184. In the illustrated implementation, velocitycalculation subsystem 184 executes a loop with blocks 222 and 224, withblock 222 collecting sensor data from one or more secondary sensors andblock 224 calculating and storing both longitudinal and lateral velocityfrom the collected sensor data. The stored velocities are then retrievedby block 220 of controlled stop subsystem 186 and used to implement thecontrolled stop operation.

It will be appreciated that implementation of a controlled stopoperation to follow a controlled stop trajectory based on sensedlongitudinal and lateral velocities of the vehicle is well within theabilities of those of ordinary skill having the benefit of the instantdisclosure. Additional sensors may also be used in connection withimplementing a controlled stop, however, so the invention is not limitedto implementing a controlled stop solely based on sensed velocity.

FIG. 3, for example, illustrates an example scene 240 where autonomousvehicle 100 is traveling along a three lane highway 242, e.g., within aright-most lane 244. Highway 242 additionally includes opposingshoulders 246, 248, and additional vehicles 250, 252 may also be in thevicinity of autonomous vehicle 100 and traveling in the same direction.In this scene, primary vehicle control system 120 may continuouslyupdate a primary trajectory 254 as well as a controlled stop trajectory256, such that if no adverse event has been detected, primary vehiclecontrol system 120 controls vehicle 100 to follow primary trajectory 254while if an adverse event is detected, a controlled stop operation isperformed to control vehicle 100 to follow controlled stop trajectory256. For each trajectory 254, 256, a subset of the control points areillustrated at 258. Thus, for primary trajectory 254, primary vehiclecontrol system 120 may define a substantially straight having asubstantially constant longitudinal velocity of about 88 feet per second(fps) and a lateral velocity of about 0 fps. On the other hand, due tothe closer proximity of shoulder 248 to lane 244, as well as thepresence of vehicles 250, 252, primary vehicle control system 120 maydefine a path for controlled stop trajectory 256 that smoothlydecelerates vehicle 100 while directing vehicle 100 into shoulder 248 tocome to rest at the position illustrated at 100′. In each case, therespect vehicle control system 120, 160 may control direction control112, powertrain control 114 and brake control 116 to appropriate matchthe velocities defined by the respective trajectories 254, 256.

Returning to FIG. 1, various types of sensors may be used as anadditional sensor from which to determine a redundant lateral velocityfor use in controlling a vehicle to implement a controlled stop. Forexample, an optical sensor, e.g., array sensor 176 (discussed in greaterdetail below), or even a non-array optical sensor, may be used tocapture images of a ground or driving surface and through image analysisdetect movement of the vehicle relative thereto. Alternatively, in someimplementations, a RADAR sensor, e.g., similar to RADAR sensor 134 inprimary vehicle control system 120, may be used as an additional sensor.In some implementations, a RADAR sensor may be a short range Dopplersensor and/or a ground penetrating RADAR sensor.

In some implementations, an optical or RADAR sensor may be oriented at anon-orthogonal angle relative to vertical and in a direction generallyparallel to a lateral axis of the vehicle in order to measure lateralvelocity along the lateral axis. In some instances, such a sensor may bedisposed on the undercarriage of the vehicle, although the invention isnot so limited, as other locations and orientations that enable velocityto be measured generally along the lateral axis of the vehicle may beused. In some instances, an optical or RADAR sensor may be exclusive tothe secondary vehicle control system, and in other instances may beshared by the primary and secondary vehicle control systems.

Other sensors capable of sensing or otherwise estimating lateralvelocity may also be used. For example, LIDAR sensors, radar sensors,sonar sensors, mechanical contact (caster) sensors, etc. may also beused in other implementations. Further, as noted above, such sensors maybe exclusive to a secondary vehicle control system or shared betweenprimary and secondary vehicle control systems in differentimplementations.

Other variations will be appreciated by those of ordinary skill havingthe benefit of the instant disclosure.

Downwardly-Directed Optical Array Sensor

As noted above, in some implementations, an additional sensor used togenerate a lateral velocity for use in performing controlled stops mayinclude one or more downwardly-directed optical sensors configured tocapture images of a ground or driving surface to sense movement of thevehicle relative to the driving surface. In such implementations, aplurality of images of the ground surface may be captured from the oneor more optical sensors, and the plurality of images may be processed todetermine the lateral velocity of the vehicle.

In some implementations, the optical sensors may form an optical arraysensor, e.g., optical array sensor 176 of FIG. 1, with individualoptical sensors arranged to have different, positionally offset, fieldsof view. For example, FIGS. 4-5 illustrate one example implementation ofan optical array sensor 300 including three optical sensors 302, 304,306 having respective fields of view 308, 310, 312 arranged generallyalong an axis A, which may correspond generally to a longitudinal axisof a vehicle, the underside of which is illustrated at 314. Fields ofview 308, 310 cover a portion of a ground or driving surface 316, and itwill be appreciated that each optical sensor 302, 304, 306 isdownwardly-directed, with optical sensor 304 generally facing along avertical axis V relative to ground surface 316, and optical sensors 302and 306 generally facing at an angle relative to the vertical axis.

Each optical sensor 302, 304, 306 may incorporate various types ofelectromagnetic radiation sensors capable of capturing an image of theground surface such that images captured at different times may becorrelated with one another to determine a positional displacement ofthe vehicle relative to the ground surface between those differenttimes. For example, an optical sensor may incorporate an image capturedevice such as used in a digital camera. In addition, an optical sensormay be sensitive to different ranges of electromagnetic frequencies,e.g., within the visible light spectrum, below the visible lightspectrum (e.g., infrared frequencies) and/or above the visible lightspectrum (e.g., x-ray or ultraviolet frequencies). Other types ofoptical sensors suitable for capturing images of the ground surface maybe used in other implementations.

As illustrated in FIGS. 4-5, in some implementations, the fields of view302, 304, 306 may be overlapping with one another to aid in correlatingthe images from optical sensors 302, 304, 306. By aligning these fieldsof view generally along the longitudinal axis A, images captured fromdifferent optical sensors 302, 304, 306 may be correlated at differentvelocities in order to determine positional displacement of vehicle 314over a time range, from which a vehicle velocity may be calculated. Atlower speeds, for example, two images captured by the same opticalsensor 302, 304, 306 may be correlated to determine a velocity, while athigher speeds the vehicle may have traveled too far for the same sectionof ground surface to be detected in two images from the same opticalsensor 302, 304, 306, such that images of the same section of groundsurface 316 captured at two different times from two positionallyseparated optical sensors 302, 304, 306 may be correlated in order todetermine the positional displacement of the vehicle.

Further, in some implementations, images from multiple optical sensors302, 304, 306 may be stitched together into composite images to providethe greater effective aperture, thereby facilitating correlating theposition of the vehicle relative to the ground surface for the purposesof velocity calculations.

Correlation of different images, in this regard, may be considered torefer to an operation by which two images capturing overlapping views ofthe same section of ground surface may be aligned with one another.Various image processing techniques that shift and/or distort one ormore images to find a positional displacement between the images may beused to perform a correlation. Where images being correlated arecaptured at different times, the correlation may be used to determine apositional displacement of the vehicle relative to the ground surfacebetween the two points in time. When images being correlated arecaptured at the same time, the correlation may be used to stitch theimages together into a single composite image. In many instances, thecorrelation relies on matching shapes or other distinctive features inthe images, as will be appreciated by those of ordinary skill having thebenefit of the instant disclosure.

FIGS. 6A-6D, for example, illustrate example scenarios whereby imagescaptured by different optical sensors 302, 304, 306 may be correlated atdifferent vehicle velocities. FIG. 6A, in particular, illustrates fieldsof view 308, 310, 312 of optical sensors 302, 304, 306 relative toground surface 316 at a first time t₀. To facilitate the discussion, aset of distinctive features 318 is illustrated on ground surface 316, itbeing understood that such shapes would ordinarily not be found on atypical driving surface.

FIGS. 6B-6D respectively illustrate fields of view 308, 310, 312 ofoptical sensors 302, 304, 306 relative to ground surface 316 at a secondtime t₁, but assuming that vehicle 314 is traveling at three differentspeeds. FIG. 6B, for example, illustrates vehicle 314 movement at arelatively low speed, such that distinctive features 318 have moved tothe position illustrated at 318′, resulting in a positional displacementrepresented by vector V₁. Vector V₁ has a direction and a magnitude, andthe longitudinal and lateral components of vector V₁ have magnitudes ordistances representing longitudinal position displacement d_(long,1) andlateral position displacement d_(lat,1) between times t₀ and t₁. Thelongitudinal and lateral velocities V_(long) and V_(lat) may becalculated from these displacements as follows:V _(long) =d _(long,1)/(t ₁ −t ₀)  (1)V _(lat) =d _(lat,1)/(t ₁ −t ₀)  (2)

Moreover, it will be appreciated that based upon the magnitude ofpositional displacement illustrated in FIG. 6B, the images captured byoptical sensor 302 at times t₀ and t₁ and represented by field of view308 may be used to determine the positional displacement between thesetimes.

At a higher speed, e.g., as illustrated in FIG. 6C, vehicle 314 movementat the relatively higher speed results in distinctive features 318moving to the position illustrated at 318″, and having a positionaldisplacement represented by vector V₂. Vector V₂ has a direction and amagnitude, and the longitudinal and lateral components of vector V₂ havemagnitudes or distances representing longitudinal position displacementd_(long,2) and lateral position displacement d_(lat,2) between times t₀and t₁. Based upon the magnitude of positional displacement illustratedin FIG. 6C, the image captured by optical sensor 304 at time t₁(represented by field of view 310) may be correlated with the imagecaptured by optical sensor 302 at time t₀ (represented by field of view308) for the purpose of determining the positional displacement betweenthese times.

Likewise, at an even higher speed, e.g., as illustrated in FIG. 6D,vehicle 314 movement at the relatively even higher speed results indistinctive features 318 moving to the position illustrated at 318′″,and having a positional displacement represented by vector V₃. Vector V₃has a direction and a magnitude, and the longitudinal and lateralcomponents of vector V₃ have magnitudes or distances representinglongitudinal position displacement d_(long,3) and lateral positiondisplacement d_(lat,3) between times t₀ and t₁. Based upon the magnitudeof positional displacement illustrated in FIG. 6D, the image captured byoptical sensor 306 at time t₁ (represented by field of view 312) may becorrelated with the image captured by optical sensor 302 at time to(represented by field of view 308) for the purpose of determining thepositional displacement between these times.

It will be appreciated that by incorporating multipledownwardly-directed optical sensors having positionally offset fields ofview, the collective field of view of an optical array sensor may belarger than that of any of the individual sensors, and may provide alarger effective aperture for the optical array sensor that enablesvelocity to be captured over a larger range and potentially using lowerframe rates. Moreover, by aligning multiple fields of view generallyalong a longitudinal axis of a vehicle, an optical array sensor may beparticularly well suited for applications where a high aspect ratioexists between longitudinal and lateral velocity, as is the case with anautonomous vehicle that may be expected to travel up to 60-100 mph ormore, but with lateral velocities that are significantly smaller, evenwhen executing turns.

While optical array sensor 300 is illustrated having three opticalsensors 302, 304, 306 oriented along a one dimensional array, theinvention is not so limited. In particular, greater or fewer opticalsensors may be arranged into an array, and moreover, arrays may bedefined in two dimensions. FIG. 7, for example, illustrates a 2×3optical array sensor 320 including six optical sensors 322, 324, 326,328, 330, 332 having respective fields of view 334, 336, 338, 340, 342,344 disposed in a 2×3 array covering a portion of ground surface 346.Moreover, in some implementations, it may be desirable to incorporateone or more strobe emitters, e.g., a flash or strobe emitter 348, whichcan illuminate the ground surface 346 in connection with capturingimages with optical sensors 322-332 to enable shorter exposure times andreduced image blurring, and thereby facilitate image correlation.

Furthermore, it will be appreciated that optical array sensors in someimplementations need not have fields of view that are precisely arrangedalong regularly spaced intervals along one or two dimensions. Forexample, the fields of view of multiple optical sensors in one dimensionmay have different spacings along an axis of the one dimension, andmoreover, may be laterally offset from one another along a transversedirection from that axis. Likewise, the fields of view of multipleoptical sensors in a two dimensional array may be separated from oneanother by different spacings.

Moreover, while optical array sensor 300 is illustrated having a singlehousing for optical sensors 302-306, in other implementations theoptical sensors of an optical array sensor may have separate housingsand may be separately mounted to a vehicle. Further, while optical arraysensor 300 is illustrated as being mounted to the undercarriage of avehicle, it will be appreciated that other mounting locations from whichthe ground surface may be imaged may be used in other implementations.As such, various combinations and orientations of downwardly-directedoptical sensors may be used for an optical array sensor in differentimplementations.

Further, an optical array sensor 300 may also include additional sensorsfor calibration purposes. For example, one or more distance sensors,e.g., laser rangefinders, may be used to sense the distance of opticalarray sensor 300 from the ground surface, such that the images collectedtherefrom and the positional displacements calculated therefrom may beappropriately scaled to compensate for a differing distance between theoptical sensor and the ground surface.

Now turning to FIG. 8, this figure illustrates an example sequence ofoperations 350 for sensing velocity using optical array sensor 300 ofFIGS. 4-5. Sequence of operations 350 may be implemented, for example,within a processor or controller of optical array sensor 300, oralternatively, within a processor or controller that is external tosensor 300, e.g., within secondary vehicle control system 160 of FIG. 1.Sequence of operations 350 is configured to sense velocity betweensuccessive time steps, which may, in some implementations, be equivalentto a control cycle of a vehicle control system, or alternatively, may beshorter or longer in duration relative thereto. For example, thecollection rate of an optical array sensor in some implementations maybe faster than the control cycle rate for a vehicle control system suchthat multiple velocities can be sensed and calculated, and in someinstances, averaged together, within a single control cycle. It isassumed for the purposes of this example that optical sensor 302 havingfield of view 308 captures “first” images, while optical sensor 304having field of view 310 captures “second” images and optical sensor 306having field of view 312 captures “third” images to represent therelative positional offsets of fields of view 308, 310 and 312 along thelongitudinal axis of the vehicle.

The sequence begins in block 352 by capturing an image with each opticalsensor (e.g., image capture device) of optical array sensor 300, andstoring the image, e.g., along with a corresponding timestamp. Controlthen passes to block 354 to determine whether this is the first timestep, and if so, passes control to block 356 to advance to the next timestep. Thus, on second and subsequent time steps, block 354 passescontrol to block 358 to select pairs of images to correlate based upon aprior velocity determination.

It will be appreciated that in many instances the maximum rate of changein velocity of a vehicle in normal operation will be relatively slow ascompared to the velocity collection rate of optical array sensor 300,and as such, in some implementations it may be desirable to reduce theprocessing overhead associated with correlating images by attempting topredict which images will likely be correlatable with one another giventhe current velocity of the vehicle and the known positional offsets ofthe fields of view 308, 310, 312. This prediction may be based in someimplementations upon storing velocity information from the prior timestep (e.g., the prior longitudinal velocity or displacement) andselecting an image based on a mapping of velocity or displacement rangesto fields of view 308, 310, 312. In other implementations, theprediction may be based upon storing an indication of which images werecorrelated in the prior time step, and reusing those images for thecurrent time step. In still other implementations, however, predictionof which images to correlate may be omitted, e.g., such that all imagesare sequentially or concurrently correlated with one another.

In addition, if no prior velocity determination has been made (e.g., onthe second time step), the first images captured by optical sensor 302at the current and prior time step may be selected for correlation inblock 358, as it may be assumed that upon startup the vehicle isexpected to not be moving.

Thus, based upon the images selected in block 358, block 360 attempts tocorrelate the images captured at the current and prior time steps, e.g.,using image processing to detect a positional offset between distinctivefeatures in the respective images. Block 362 determines whether thecorrelation was successful, and if so, passes control to block 364 tocalculate and store the aforementioned lateral and/or longitudinaldisplacements and/or velocities in the manner described above. Controlthen returns to block 354 to advance to the next time step.

If not, however, the prediction was unsuccessful, and block 362 passescontrol to block 366 to attempt to correlate all images. Thus, forexample, for optical array sensor 300, the image captured by opticalsensor 302 at the prior time step may be correlated with the imagescaptured by each of optical sensors 302, 304, 306 at the current timestep, the image captured by optical sensor 304 at the prior time stepmay be correlated with the images captured by each of optical sensors302, 304, 306 at the current time step, and the image captured byoptical sensor 306 at the prior time step may be correlated with theimages captured by each of optical sensors 302, 304, 306 at the currenttime step.

Next, block 368 determines whether any of the correlations weresuccessful, i.e., whether any pair of images could be mapped. If so,control passes to block 364 to calculate and store the aforementionedlateral and/or longitudinal displacements and/or velocities in themanner described above based upon the images that were successfullycorrelated.

If not, however, block 368 passes control to block 370 to optionally logan event indicating that no correlation could be found. Doing so, forexample, may enable subsequent time steps to ignore the current timestep and correlate between images spanning multiple prior time steps.Doing so may also enable more systemic errors to be detected, e.g., dueto failures in sensor hardware, optical sensors being blocked oroverly-soiled, etc. As such, block 370 may also signal an alert in someinstances, e.g., after N successive time steps without a successfulcorrelation. Block 370 then returns to block 354 to continue to the nexttime step.

Sequence of operations 350 therefore attempts to match different imagescollected at different points in time by different optical sensors inorder to determine a positional displacement of the vehicle over thosepoints in time, and from which at least lateral velocity may bedetermined. However, other sequences of operations may be used in otherimplementations. FIG. 9, for example, illustrates an alternativesequence of operations 380 that generates composite images from theimages of multiple optical sensors and performs a correlation betweenthe composite images at different points in time. Block 382, forexample, captures images from each of optical sensors 302-306, and block384 stitches the images into a single composite image. Block 384, forexample, may perform a correlation between each of the images to shiftand/or distort each of the images such that the overlapping fields ofview align with one another, resulting in a composite image covering allof the fields of view. Block 384 may also store the composite imagealong with a timestamp associated with the time at which the image wascaptured.

Next, block 386 determines if this is the first time step, and if so,passes control to block 388 to advance to the next time step. Thus, onsecond and subsequent time steps, block 386 passes control to block 390to attempt to correlate the current composite image with a priorcomposite image (e.g., the composite image for the immediately priortime step), e.g., using image processing to detect a positional offsetbetween distinctive features in the respective composite images. Block392 determines whether the correlation was successful, and if so, passescontrol to block 394 to calculate and store the aforementioned lateraland/or longitudinal displacements and/or velocities in the mannerdescribed above. Control then returns to block 388 to advance to thenext time step.

If not, however, block 392 passes control to block 396 to optionally logan event indicating that no correlation could be found. Doing so, forexample, may enable subsequent time steps to ignore the current timestep and correlate between images spanning multiple prior time steps.Doing so may also enable more systemic errors to be detected, e.g., dueto failures in sensor hardware, optical sensors being blocked oroverly-soiled, etc. As such, block 396 may also signal an alert in someinstances, e.g., after N successive time steps without a successfulcorrelation. Block 396 then returns to block 388 to continue to the nexttime step.

Optical array sensors of the type disclosed herein may be used, forexample, to determine one or more velocities associated with vehiclemovement, including, for example, direction and/or magnitude componentsof an overall velocity of a vehicle and/or magnitudes of velocity alongone or more predetermined directions (e.g., lateral velocities and/orlongitudinal velocities respectively along lateral and longitudinal axesof a vehicle). Moreover, in addition to or in lieu of storing one ormore velocities, an optical array sensor may also communicate, send, orotherwise output the determined displacements and/or velocities to anexternal device such as a vehicle control system. Furthermore, while anoptical array sensor is disclosed herein as being used by a secondaryvehicle control system, the invention is not so limited. In particular,in some implementations an optical array sensor as disclosed herein maybe used by a primary vehicle control system, or even in implementationswhere no secondary vehicle control system is used.

In addition, while optical array sensors as disclosed herein provide anincreased effective aperture that enables images captured at aparticular sampling rate to cover a wider range of velocities, in someimplementations the sampling rate may be dynamically adjustable to coveran even wider range of velocities. Further, in some implementationsmultiple optical sensors may have comparable fields of views to provideadditional redundancy.

Other variations will be apparent to those of ordinary skill. Therefore,the invention lies in the claims hereinafter appended.

What is claimed is:
 1. A method of autonomously operating a vehicle,comprising: receiving a plurality of images of a ground surface capturedfrom a plurality of downwardly-directed optical sensors mounted to thevehicle, each of the plurality of downwardly-directed optical sensorshaving a respective field of view, and the plurality ofdownwardly-directed optical sensors arranged such that the respectivefields of view differ from one another, wherein at least a subset of theplurality of downwardly-directed optical sensors are arranged to haverespective fields of view that are positionally offset along aone-dimensional array, and wherein the subset of downwardly-directedoptical sensors arranged to have respective fields of view that arepositionally offset along the one-dimensional array includes first andsecond downwardly-directed optical sensors; processing the plurality ofimages using at least one processor to determine a velocity of thevehicle, including: correlating a first image captured at a first timeby the first downwardly-directed optical sensor with a second imagecaptured at a second time by the second downwardly-directed opticalsensor to determine a positional displacement of the vehicle between thefirst and second times; and determining the velocity based upon thedetermined positional displacement of the vehicle between the first andsecond times; and autonomously controlling movement of the vehicle usingthe determined velocity.
 2. The method of claim 1, wherein at least asubset of the plurality of downwardly-directed optical sensors arearranged to have respective fields of view that are positionally offsetin a two-dimensional array.
 3. The method of claim 1, wherein theone-dimensional array extends generally along a longitudinal axis of thevehicle.
 4. The method of claim 1, wherein the respective fields of viewof at least a subset of the plurality of downwardly-directed opticalsensors partially overlap with one another.
 5. The method of claim 1,wherein processing the plurality of images includes: correlating thirdand fourth images respectively captured at third and fourth times by athird downwardly-directed optical sensor among the plurality ofdownwardly-directed optical sensors to determine a positionaldisplacement of the vehicle between the third and fourth times; anddetermining a second velocity based upon the determined positionaldisplacement of the vehicle between the third and fourth times.
 6. Themethod of claim 1, wherein correlating the first image captured at thefirst time by the first downwardly-directed optical sensor with thesecond image captured at a second time by the second downwardly-directedoptical sensor is a first correlation, and wherein processing theplurality of images further includes: performing a second correlationbetween the first image captured at the first time by the firstdownwardly-directed optical sensor and a third image captured at a thirdtime by the first downwardly-directed optical sensor; and using one ofthe first and second correlations to determine the positionaldisplacement of the vehicle between the first time and one of the secondand third times.
 7. The method of claim 6, wherein the second and thirdtimes are substantially the same.
 8. The method of claim 6, whereinusing the one of the first and second correlations to determine thepositional displacement includes: using the first correlation inresponse to determining that the second image could be correlated to thefirst image; and using the second correlation in response to determiningthat the third image could be correlated to the first image.
 9. Themethod of claim 1, wherein the subset of downwardly-directed opticalsensors arranged to have respective fields of view that are positionallyoffset along the one-dimensional array includes more than twodownwardly-directed optical sensors.
 10. The method of claim 1, whereinthe one-dimensional array extends generally along a longitudinal axis ofthe vehicle, and wherein the subset of downwardly-directed opticalsensors arranged to have respective fields of view that are positionallyoffset along the one-dimensional array collectively provide an effectiveaperture that is larger along the longitudinal axis of the vehicle thanalong a lateral axis of the vehicle.
 11. The method of claim 1, whereinthe plurality of images includes images captured at a plurality of timesby each of the subset of downwardly-directed optical sensors.
 12. Themethod of claim 11, wherein processing the plurality of images includes:stitching together multiple images from each of the plurality of timesto generate a composite image for each of the plurality of times;correlating a first composite image for a third time among the pluralityof times with a second composite image for a fourth time among theplurality of times to determine a positional displacement of the vehiclebetween the third and fourth times; and determining the a secondvelocity based upon the determined positional displacement of thevehicle between the third and fourth times.
 13. The method of claim 12,wherein the respective fields of view of the subset ofdownwardly-directed optical sensors partially overlap with one another,and wherein stitching together the multiple images from each of theplurality of times includes correlating overlapping regions of themultiple images.
 14. The method of claim 11, wherein the plurality oftimes are within a single control cycle among a plurality of controlcycles.
 15. The method of claim 1, wherein the plurality ofdownwardly-directed optical sensors are coupled to a primary vehiclecontrol system of the vehicle, wherein the primary vehicle controlsystem includes the at least one processor, and wherein receiving theplurality of images, processing the plurality of images, andautonomously controlling movement of the vehicle are performed by theprimary vehicle control system.
 16. The method of claim 1, wherein theplurality of downwardly-directed optical sensors are coupled to asecondary vehicle control system of the vehicle, wherein the secondaryvehicle control system includes the at least one processor, and whereinreceiving the plurality of images, processing the plurality of images,and autonomously controlling movement of the vehicle are performed bythe second vehicle control system subsequent to detection of an adverseevent for a primary vehicle control system of the vehicle.
 17. Themethod of claim 1, wherein the plurality of downwardly-directed opticalsensors are disposed on an undercarriage of the vehicle.
 18. The methodof claim 1, wherein determining the velocity includes determining alateral velocity of the vehicle.
 19. The method of claim 1, whereindetermining the velocity includes determining a longitudinal velocity ofthe vehicle.
 20. The method of claim 1, wherein determining the velocityincludes determining a magnitude component and a direction component ofthe velocity.
 21. The method of claim 1, wherein the vehicle comprises afully autonomous vehicle.
 22. The method of claim 1, wherein the vehiclecomprises an autonomous wheeled vehicle.
 23. The method of claim 1,wherein the vehicle comprises an autonomous automobile, bus or truck.24. The method of claim 1, wherein the plurality of downwardly-directedoptical sensors are infrared sensors.
 25. The method of claim 1, furthercomprising: capturing the plurality of images with the plurality ofdownwardly-directed optical sensors; and illuminating the ground surfacewith a strobe emitter when capturing the plurality of images.
 26. Avehicle control system, comprising: a plurality of downwardly-directedoptical sensors configured to be mounted to a vehicle to capture imagesof a ground surface, each of the plurality of downwardly-directedoptical sensors having a respective field of view, and the plurality ofdownwardly-directed optical sensors arranged such that the respectivefields of view differ from one another, wherein at least a subset of theplurality of downwardly-directed optical sensors are arranged to haverespective fields of view that are positionally offset along aone-dimensional array, and wherein the subset of downwardly-directedoptical sensors arranged to have respective fields of view that arepositionally offset along the one-dimensional array includes first andsecond downwardly-directed optical sensors; and at least one processorcoupled to the plurality of downwardly-directed optical sensors, the atleast one processor configured to receive a plurality of images of theground surface captured from the plurality of downwardly-directedoptical sensors, process the plurality of images to determine a velocityof the vehicle, and autonomously control movement of the vehicle usingthe determined velocity, wherein the at least one processor isconfigured to process the plurality of images by correlating a firstimage captured at a first time by the first downwardly-directed opticalsensor with a second image captured at a second time by the seconddownwardly-directed optical sensor to determine a positionaldisplacement of the vehicle between the first and second times anddetermining the velocity based upon the determined positionaldisplacement of the vehicle between the first and second times.
 27. Avehicle, comprising: a plurality of downwardly-directed optical sensorsconfigured to be mounted to a vehicle to capture images of a groundsurface, each of the plurality of downwardly-directed optical sensorshaving a respective field of view, and the plurality ofdownwardly-directed optical sensors arranged such that the respectivefields of view differ from one another, wherein at least a subset of theplurality of downwardly-directed optical sensors are arranged to haverespective fields of view that are positionally offset along aone-dimensional array, and wherein the subset of downwardly-directedoptical sensors arranged to have respective fields of view that arepositionally offset along the one-dimensional array includes first andsecond downwardly-directed optical sensors; and at least one processorcoupled to the plurality of downwardly-directed optical sensors, the atleast one processor configured to receive a plurality of images of theground surface captured from the plurality of downwardly-directedoptical sensors, process the plurality of images to determine a velocityof the vehicle, and autonomously control movement of the vehicle usingthe determined velocity, wherein the at least one processor isconfigured to process the plurality of images by correlating a firstimage captured at a first time by the first downwardly-directed opticalsensor with a second image captured at a second time by the seconddownwardly-directed optical sensor to determine a positionaldisplacement of the vehicle between the first and second times anddetermining the velocity based upon the determined positionaldisplacement of the vehicle between the first and second times.
 28. Anon-transitory computer readable storage medium storing computerinstructions executable by one or more processors to perform a method ofautonomously operating a vehicle, comprising: receiving a plurality ofimages of a ground surface captured from a plurality ofdownwardly-directed optical sensors mounted to the vehicle, each of theplurality of downwardly-directed optical sensors having a respectivefield of view, and the plurality of downwardly-directed optical sensorsarranged such that the respective fields of view differ from oneanother, wherein at least a subset of the plurality ofdownwardly-directed optical sensors are arranged to have respectivefields of view that are positionally offset along a one-dimensionalarray, and wherein the subset of downwardly-directed optical sensorsarranged to have respective fields of view that are positionally offsetalong the one-dimensional array includes first and seconddownwardly-directed optical sensors; processing the plurality of imagesusing at least one processor to determine a velocity of the vehicle,including correlating a first image captured at a first time by thefirst downwardly-directed optical sensor with a second image captured ata second time by the second downwardly-directed optical sensor todetermine a positional displacement of the vehicle between the first andsecond times and determining the velocity based upon the determinedpositional displacement of the vehicle between the first and secondtimes; and autonomously controlling movement of the vehicle using thedetermined velocity.